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In its comparatively brief history, the field of measuremeWt and 
assessment has been developed progressively to provide more adequate 
descriptions of both Individuals and yroups. Measurement of the 
characteristics of individuals has, in fact, captured a- far greater 
share of the.^'attention of educators and psychometricians than has the 
measurement of group characteristics. To an increasing extent, however, 
the measures -used to provide information about indi-viduals are designed 
f or* and ^administered uniformly to groups of examinees and the scorR are 
reported uniformly to all members of the group. Little attention is 
yiven to the characteristics' of individual performance beyond the bare 
fact of a scaled score, usually expressed in relation to a group norm. ^ 
f believe we are unintentionally and unconsciously missing a great 
deal of important information about the individual through the conventional 
process that I.have just described. Not only are we ^itiissing that information 

we being those who develop and use the tests but, more importantly, 
the individual examinees themselves are missing information that would be 
satisfying and useful to, them. The deficiencies are.-most serious, I 
believe, when those individuals are engaged in non-formal education as 
contrastisd with education in. the more conventional school -cl assroom 
settings- that have dominated our thinking in the designing of instruments 
and programs. I shall illustrate my concern by talking mainly about two 
areas of current work at ETS and elsewhere: 

first , an attempt to develop and apply techniques^ to assess the 

educational value of life experiences out-of-school experiences 

• • • 

« 

•-of individuals, especially adults contempl ating new. educational or 
work experiences. 



second ^n attanpt to provide more complete and analyt^al Information 
to the^ examinee abput the nature and quality of his or her test 
performance than is typically convej^ed in a numerical score report. 
This latter effort is applicable whether one isilooking at performance 
in rton- formal or formal, settinys.. 

Non^traditiooal Study in the U.S. 

In introcTuciny. the 'topic of assessing thejndi vidual ' in non-traditiona 
study, I should perhaps mentionjthat concern with the topic of the older 
student both on *and off campus is strong and fajrly new in my own country. 
The changing/atteVn^of interest in higher education iq the United States* 
is illustrated by it\e fact that colleges and universities h'ave increased 
their enrollment of students over 35 years old by more than one-third 
over the past five years.* And many njore of these students who are 
"older" by traditional standards are' involved 1n non-formal education 
rather than reg'ul arly .enrol 1 ed. Let me qiiote from an article by^Peter 
Urucker that appealed this month" in The Qhronicle of Higher -Education , a 
publication widely read .in the United Stkes. It illustrates the fact^ 
that non-traditional or non-formal -.educafti on is a preoccupation not only 
in' many countries that have been called underdeveloped but in some trhat 
have Deen called overdeveloped as well *\ 

' *The Chronicle of Higher Education , May 4, 1981, p.3 (Article by Jacl^ 
Magarrell, quoting data from U.S. Census Bureau) r . * 



Drucker's article says: • 

"Demand for educa,tion is actually going up,, not down. What is 
going down, and fairly fast, is demand for traditional education In 
traditional schodls.- " . * 

"Indeefl, the fastest-growing industry in America today may be the 
continuing professional education of, highly schooled midcareer 
adults. Much of it takes place outside the education establishment ' ^ . 
— through companies, hospitals, and govprnmentybureaus that run 
courses for managerial and professional employeesi or through- 
management associations and trade associations. Meanwhile, any 
number of private entrepreneurs are organizing seminars and courses^, ^ 

producing training films and audiotapes, and otherwise taking 

/ 

ad^anta^e of growth opportunities that university faculties shy away 
from. 

* 

"The demand for continuing education does not take the form that 
most observers, including this writer, 'originally expected namely 
'Great Books' classes for adults wanting to learn about the,^humanities , 
the arts, the 'life of them'ind.' We face instead an all but insatiable 
demand for advanced professional education; *in engineering and ^ 
medicine, in -accounting and journalism, in law, and in administration 
and ipanayement. 

^'Yet the mature adults who come back for such studies also demand 
what teachers of professional subjects, are so rarely able to supply: 
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a humanistic perspective that can integrate' advanced professibnal and 
t^hnical knowledge into' a broader universe of experience and learning. 
Since these new students' a1 so need unconventional hou-rs evenings, 
weekends, or h1g|i-intensity courses^ that cram a semester's work into 
two weeks their demands for learning pose a vague but real threat 
to academe. Acadene's standard response producing new Ph.D.'s for 
a new -'department' is roughly comparable to restyling t|ie buggy 
whip for leadership in the new market for 'horseless carriages.' 

. "The greatest challenge to educators is likely to come from our new 
opportunities for diversity. We now havd the chance to apply the 
basic findings of psychological, developmental, and educational 
research over the last 100 years; namely, that no one. educational 
method fits all chlldHPefi." / • • ^ ^ 

And, of course, n&ither does any one method of either education 
or assessment fit all aduUs. And this brings us back tb the fir.st 
area I would 'like to discuss: the assessment of life experience or 
non-traditional study. ^ ' 

1. Assessment in Non-Traditional Study 

To illustrate an approach to this area assessment of life 
experiences or non-traditional study — a little history of some 
work that we at ETS have been engaged in over the past two or three 
decades may be useful. _ ^ • 

During the^l960s my colleagues and I had been watching the emergence 
.of non-traditiorial study, among pther innovative trends and developments 

0 
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•T ideas .that in many ca;ses required new forms of ^ses^sment if they 
were to><o'H^, Our job is to work with the'teaching institutions to 
devise assessmpatrprocedures that fit the n?w way's' f)f teaching and 
learning that grow rigfft out of ttiein rattier tfian fitting awkwardly 
or in an^ way constraining the educational process itself. '^Unless 
that is done,- the forms of i.ieasurement can ha>e a deadening effect 

on educational innovatvon. Nothing could be more unfortunate. It 

< 

is imperative that asse'Ssment methojjs^.be found that will encourage 

* r . • 

flexibility rather than, impede it. 

Because of that belief, ETS had worked for. many years to explore ^ 
with' the teaching institutions a variety of ways in which competency 
can be measured without any- presctiption as to the natur? of the 
learning process that brought it about. Eleven years d-go, ETS 
became especially active in the whole field of non-traditional stutly 
through stimulating the formation of a nationwide' Commi ssion on 
tlon-Traditional Study which did jts^work^n the early i970s. 

As that cdimi-ssion explored the potential of the non-traditional 
.movement, it recognized that assessment is a critical and underdeveloped 
aspectf of unconventional ^programs. Indeed, at the time I said "I 
see assessment as the key to the success o-r failure of the whole ' ^ 
^non-traditional experiment, because the assessment procedures must ♦ 
be strong enough to satisfy and infonn the student and the teacher, 
and strong enough too to carry what I call 'third p^arty credibiHty.* 
This is what it so often lacks, but what it must achieve. 



"Assessment -in experiential learning- is likely to make relatively 
slight use of anything recognizable as tests, I suspect* Rather, it 
may take the form of evaluative procedures described ifi some detail" 
and followed carefully, so that the ^absent third party has a basis • 
.for reposing confidence in the equity and relevance of the result. 
If_^we can together devise such procedures -y^a red intimately to the 
underlying educational experiences, make them explicit, and see 
that they gain vSidespread understanding, the cause of educational 
' flexibility bised on recognition of competency. will be well served 
and the purposes of ETS^^win have been fulfilled." 

And indeed we found that the assessment could be accomplished with 
some rigor in ways that hundreds of instituti'ons are now applying, 
especially with students who are based off-campus. Some of the 
techniques are anything but new: essay exams, performance tifests, 
interviews, ratings, objective written j:ests. Others involve 
simulations, assessment centers, and product assessment, self-assessment, 
and the assembly of "portfolios" in which the individual could 
* collect analyze and display his or her accomplishments. Our yoli 
was to find ways to adapt t'tiese methods to nontraditional education 
with sufficient rel iabil ity and validity to serve as-a solid base 
tor assessment and the awarding of cr>edit. The widespread acceptance 
of resulting techniques suggests that faculties and administrators 
find, them helpful . > 

The point I am making in this bri?l^ecital of the methods we 
plored or 'developed is that irr flon-traditional education the assessment 
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•method selected ,^nd the p^rticul ar data"yathered are c*sen with an . 
overall educational perspective but in sach a way as to* maximize attention 
to the special accomplishment of the individual along dimensiofis unique 
to him or her. We must, however, be able tb certify to the equivalency, 
with regard to "important and common educat-ional objectives, of h;ghly 
' disparate experiences of different individuals. I do not believe we know 
"^how to do so with the precision that can be brought to the equating -of . 

scores on parallel tests, for example, but I think the effort was quite 
• successful In achieving a useful level of stability and comparability". 
But I do believe that deriving the specific nature and diiflensions of 
accomplishment to be assessed from the unique 'experiences qf^he individual 
is fundamental if we are to do justice to individual assessment in - 
contrasVto common practice th"^ increasingly requires testing o'f all 
.individuals on the same dimensions and on common group measures. ^— -7^- 

The area of exptirientiaV learning is now being looked at with 
special intensity, in the United Stat'es at least, in the case of women 
who are often termed "non-working" because nobody is paying them: mothers 
doing the work of the home. Listen, to the words ofjir^ee Hechinger,. • 
appearing in Newsweek, May 11, under .the" ifonic, titl e" "Happy ^Mother" s 

Day.".: ' ' . ■ ' * . ' 

"Mother's job ha^ no status in a society -which rewards the 
single-minded pursuit' of money ^and success... 

"Furtheniiore,^her has no cre4€flt+4f^.' She is an unpaid amateur, 
anl^this gives her low status in a society ^pbsessed with profess i-onal , 
certified expertise. Whatever its past benefits, the professional ization' 



oT chn4 care Iras had the demeaning side effect of reducing mother • 
to a bewildered consumer of expert (and often contradictory) advise. 
The glory has yone -tjO an ever^creasing army of experts — pediatricians 
psycho! ogist.s and socia! workers — who have defined her job in 
terms ;^ their^own specia! doctrines and p/-ejudices. ttother no 
lonyeF-knows best. ^ 

"B^ause it is done free and done by women, mother's work should not 
be devalued. But we a!! Know it is. Gad our society ever respect a 
servKe freely yjven? ^ * - 

"Although nearly everyone feels qualified to *judge other people's 

children, there are no effective yardsticks by which to measure the 

quality of a mother's work. #n<l there is no way of placing a 

monetary value Qn the art of bringing up children or of transferring 

II • 

' skills needed to raise children to the ' outside' world of work. 

* • 

lirace Hechinger.'s concern is apparent and basically correct. I 
thought, however, that you would'be interested in knowing -of the work of 
Ruth Ekstrom of the ETS staff, whose specific; interest is in remedying, 
the lack that Mrs. Hechinger finds: i.e., finding ways of transferring 
skills learned and practiced by women vyorkiny in the home to the outside 

world of work.- • 

Ur." Ekstrom' s research' is only now becoming knowli. Let me quote a 
brief non-technical accou'nt that appeared three months ayo in The irjashington 
Post , by wr'iter Carol Krucqff. unider the title "CAREERS: Making Life-Work 
Experiences PayUff.":* 

10 



'"Barbara Sudler l.anded a job as direct^i^of Historic Denver, Inc., 
•on the strength of admini strati ve^^expjerience from homemakiny and 
volunteer work. 

•^Elizabeth Mohr drew on 16*years of volunteer experience — four^with 
the Atlanta Public Library ~ to be come its public information 
officer. * * 

"Patricia Bashaw combined paid work experiencfe as a teacher and 
.volunteer work skills to^become an industrial-relations representative 
for Memorex in California, where she runs training and management- 
development programs.* 

"Thesa three success .stories are cited in a new workbook for women 
who want to change careerjf, enter or return to paid work after 
spending time as homeinakers and\olunteers. iTis designed to help 
identify skills learned outside^the workforce and match them with 
paid jobs. 

"Ju^t as hundreds of colleges ndw^ive academic credit for life-work 
experience,' says author Ruth Ekstrom^, *we want employers tcgive^ 
womfen credit for the skills they have developed as homemakeri, 
mothers, community organizers and^luntejer workers." 

Ek^trom, a'researcher at Educational Testing Service in New Jersey, 
says "many women have trouble finding- work or settle for being 
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underemployed because tfiey underrate and undervalue their pwn 
ex(ierience. 

"A woman may- say, 'Oh, I've just done a little volunteer work for . * 
the Sierra Club,! when, in fact, she may have Investigated the • 
'environmental impact Of something, reviewed legislation, drafted a 
sample bill and lobbied for passage. Those are highly ^narketable 
SkilVs. 

"Or they wil^ focus on the setting in which they worked, rather than 
the skills they learned. ^ For example, if a woman says she volunteered 
for the Red Cross, you imagine, her toting bedpans. Wh^it-sha didn,*t 
say, but should have, is that she headed a muUi-mi VI ion-doll ar 
fund-raising campaign. . . ^ - » 

r * 

"Women also tend to narrdw their sights to- traditional ,'women"s 
jobs," she says. "But some of the best opportunities are in non- 
traditional work, where employers are actively seekiny^women, and, 
the pay is much better. • - ' 

i 

:'A woman who's been working on the family car should consider' 
mechanics. If a secretary i^ great at fixing the office copier, she 

- . • * 

should look into of f ice -machine repair. Someone who's got a green 
thumb may get a- job, with a company that maintains plants, in office 
building^." . ^ - 

12 
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The new workbook was developed as pan of Project HAVE (Homemaki ng 

and Volunteer) Skills, supported by.^a grant from the United States 

■ Department Educai.ion. .^u.r.rent1y, i s being u*d' and evaluated 

•by several dozen- women,' s' centers^, s-ays Ekstrom, who hope? a revised 
' ' • • • 

version will be available to thfe public next fall.' ; 

Her> advic.e for re-entry women, or t'hose wlio want to change Careers: 

l .i>.l ist all work you've done / Think back over everything 
■ybu>e done as a parent", volunteer and comrtiunit^;jfte^ber and include 
everything from planning and cooking meals to serving on tenants, 
groups'Vr--wafl papering the bathroom. Don.'.t forget education and 

paid work. ' , ^ 

2: List ^the" thing.s .people say vou.'re good at . Dop^swbake 
th« be^t/ beans irvtown/have a perfectly balaflced checkbook, 1 i.sten 
well and give gdbd acivice? 
^ 3. Group items on your list into' related categories . Balancing 
the family budget, preparing your tax return and serving 'as treasurer. 
■ of the PTA alTinvol ve'financial management. "Some oWer categories 
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ma^be: health care, public rel ations/communicalJions, teaching, arts/ 

■ crafts sales/ fund- raising, clothing/ttxtil es, mechanical /technical ." 

• Those are steps that Dr. Ekstrom advocates for women before investigating 
•job possifii'l-ities, 'whither they plan to apply on their own or use the _ 

services of an occupational counselor. The point to be "emptiasized is 
Ithat the Ek-Strom approach derives the dimensions of accompr^shment to be 
assessed from the i'nterests and background of the individual rather than 
from a predetermined>list of charactertstics to be measured. 
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2/ More Complete Irrformat'ion to the Student 

Another area in which the individual has. often seemed submerged 

within thfe group is in the reporting^of test scores. For som^e time 

I hidve felt that more complete information can and should be provided 
' * - * • 

to the indivi<lual.*5tudent, and increasingly we are incorpo!;ating,M*n 
the score report material, going to each student, inf9rmation about 
' how that individual performed or^ the test as well as the score 'that ' 
he or she obtained. > - ' - 



There is nothing new, of course, about diagnosticT^scoring and 

reporting: the procedures are routine in meftiy testing services and 
* 

programs^ especially those Intended to s^ve instructional purposes 
in the classroom. It has not been customary, however, at least in 
the'United States, to provide detailed iAformation to the studen^ in 
the case of large-scale nationwide external testing 'programs .iwtt^jded 



orimaf lUl for administrative- purposes* especially ^flection for 



afl^^r 
sglilPo 



cdlleglHpror scholarship awards. 

\ ' . . ■ • . 

A fd^irly straightforward exartiple is the. case o^f the Preliminary. 
Scholastic Aptii*ude 'Test'of -the College Board, used i-n p^ as a 
component of the National Merit Scholarship Qual ifyinjj' program. For 
the J^SAT we used to report to e^ch student simply ttie two scores 
obtained*-- verbal and mathematical as scaled scores, together 
*with norms tables. Since the, program is a large *one, given to about , 
1,300,000 students each year and reported ox) ^ tight schedule, t1>e 
bare bones report of individual scaled score perfomnance together 
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with a general interpretive booklet — the saaie for. everyone — 

.explaining the nature of the test seemed to be about the b>st we 

could do. ' ^ . 

.Last year, however, we found we could do'^ littje more. The new 
system adopted j'n October, 1980 returns\he test book and keyed ^ 
answers to the school and the student. The individual report, shows 
the student not only the answers keyed aS' correct but also the 

^^student's raw scores as well as*.sca1ed scores, the way the Tatter 
are computed, the conversion tables used, and the reasons for using 
scaled rather than raw scores" irr most reporting. It is a simple 
^matter to have the computer. tal ly and reADrt to each student the 
number ,of items that he op she got rigff, number wroijg, ftumber 
omitted and number not reached, and to break those numbers down 
according to type of question, content of question and difficulty 
level. « / < , / 

The expanded score reports of course require more interpretive 
material to help the students understand the derivation and possible 
5jgnifican^ of the additional data, 'and there wer,e some -who feared 
we Would be overwhelming them. By means of ai questionnaire;and . 
student interviews, however, we found that the additional de*ftil was 
extremely popular with counselors and with the intended primary 
audience, the students. They seemed to feel that we had taken the"* 
trouble to try to answer their very basic but very general question: 
"How did I do?'' 
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These steps toward, rtcher Information far individual students merely 
jscrat'c^ the surface, of course. More of the narrativer statements 
clarifying individual- performance can be printed^ the computer on 
each jndividuarl report and keyed to the constellation of performance 
indicelS* r^l ated to the particular student, s work product the 
responds on the -answer sheet. Such verbal reports and i-nterpretive 
statements are beings used in smaller programs. 

One reason for my interest in seeing testing or assessment programs'" 

— even the largest, and perhaps especially the largest — stress more 
— »- ^ ^ ^ 

complete information to the individual is- my long-standing concern with 
what I havfe tailed "student centered assessment." , 

In a 197b paper entitled "Power to the Person" I noted that the 
titfa was not dra^n firan any social or political movement but rather that 
it grew'"... frm the^iew that education must be person-centered, that 
the critical .choices fn education, a^^n^^all things, are best made by the 
individual wlio has to 1 fve Vith''tfte% a^ that the results of measurement 
can and should be used to help learners^ taike charge of their learning and 
hence of their 1 ives. " 

J'The'main job of parents is to make themselves unnecessary, and 
the main job of the schools is^ to help devel9p self-sufficient people ... 
The systematic development of a person's ability to manage effectively 
and exercise his Or her i ndividual ity ' . . .[is]. . . a central responsibility 
of education." ' ' ^ 

V 

s ' * 

*irTS Annual Report, Educational Testing Service, 1976. 
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If this goal is to be realized,, it fs^clear that the results' of assessment 
.in learning, whether forma} or non-formal, must be cofrmiunicated cl^e^rTy 
to all students, together with the aids necessary for the interpretation 
of those results. 

There is no time in, this pap§r^to explore other exciting prospects 
for improving our assessment; of the individual in a system shaped and 
daninated by^group processes. One of the most prpmising avenues is seen 
In thegrowth of computer-assisted testing, variously called by 9ther names 
such as adaptive testing or tail ored- testing , in which the student's 
per^formance on a series of Items determines the selection of the next 
item to be presented. My- purpose today, however, is to lay stress not oa 
the pDS^tbil ities that arfe available in an ^automated society, which may 
be'^closer to or farther from attainrfent in different parts -of "Trhe world, 
but to speak of techniques that minimize the need for high technology^ 
are readiVy available, and are now being applied. ^ 
— Any adequate discussion of thfe broad topic implied by the tit^e 
of my paper would yive equal time to the special newels of group measurement, 
defining the group as a class or i wUole country or anything i/i between. 

suffer much from the fact that typically we apply measures designed 
■for individual assessment to problems in which the real need is for a set 
of'descriptors of group performjince — a task which could be performed 
more fully and much more. efficiently through matrix sampling or other 
techniques designed .specifically for-the purpose. , . 

The" primary measijrement and evaluation issue to be addressed 
today, however, is the 'need^r special attention to improved assessment 
of the individual. That's almost always the assessment i*ssue that 
needs first attention, in education beth formal and non- formal. 



